A Framework of Construct-Irrelevant Variance for Contextualized Constructed Response Assessment

Estimating and monitoring the construct-irrelevant variance (CIV) is of significant importance to validity, especially for constructed response assessments with rich contextualized information. To examine CIV in assessments, we developed a framework including model accounting measurement that could differentiate CIV. Specifically, includes due three factors: variability assessment item scenarios, judging severity, rater scoring sensitivity scenarios tasks. We proposed using many-facet Rasch (MFRM) because this can compare different factors on shared scale. demonstrate how apply framework, applied video-based science teacher pedagogical content knowledge (PCK) assessment, two tasks, each scenarios. Results task I, which assessed teachers’ analysis student thinking , indicate was substantial, while severity responses not. For II, responsive teaching results showed all substantial. discuss conceptual methodological contributions, inform development.